Various types of data have a judgment label associated with data item. For example, query-URL pairs are labeled with relevance data by human judges that indicate how relevant the URL (document) is to that query. The labeled data may then be used in training web search ranking models. In general, the better the labeling, the better the ranking model.
From time to time such data needs to be re-judged with respect to how it is labeled. For example, URL documents and content change over time, and a fresh judgment needs to be made every so often so as to re-label that document when appropriate. However, re-judging/relabeling operations are expensive, as it involves significant amounts of data being judged by human judges in order to obtain enough labeled data (to perform practical training, for example).